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(57) Abstract: Disclosed is a system that permits 
a variable number of disparate streams of datato be 
packaged together as content into a single distinct 
program referred to as anomni media package. A 
framework definition may be specified and created 
for theomnimedia package to allow a client set top 
box to decode the information and provideaccess 
to streams comprising video, audio and metadata 
information. The frameworkdefinition may be 
employed by a broadcast system to create a 
broadcast stream. Amechanism is described that 
permits a content provider to define a framework 
for delivering a package of related content. The 
framework definition encapsulates information 
necessary to build, format, transmit and display 
the disparate contents treams. Data may be 
downloaded to a receiver prior to the broadcast 
of an associatedprogram. The present invention 
may be implemented on terrestrial, cable, satellite, 
VDSL and other transport systems, including 
those that support upstream communication. 
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SYSTEM AND METHOD FOR PROVIDING AN OMNIMEDIA PACKAGE 
Cross Reference to Related Applications 

5 

This application claims the benefit of U.S. patent application number .60/253, 168 entitled 
'OmniMedia Package", filed November 27, 2000 by Steven Reynolds, which is also 
specifically incorporated herein by reference for all that it discloses and teaches. 

10 Background of the Invention 

a. Field of the Invention 

The present invention relates to enhanced multimedia television and more 
particularly to a system and method for organization, combination, transmission and 
15 reception of media from a range of sources wherein the media may comprise a plurality 
of video streams, audio streams, and other information, such as may be accessed via the - 
Internet. 

b. Description of the Background 

20 The fonnat of television programs often conforms to NTSC, PAL, or SECAM 

standards wherein a predefined bandwidth, i.e. channel, is employed to carry a single 
television program. Additional information that may be provided with a program has 
often been encoded into the vertical blanking interval (VBI), such as closed captioning or 
alternate language support, for example,. As television broadcast formats move to digital 

25 transmission, many programs continue to be presented in a manner similar to 'channel' . 
based television, comprising one video stream, a primary audio stream, and possibly an 
alternate audio stream. 

Continued expansion of the Internet and high bandwidth networks provides afceess 
to ari increasing, volume of information. Adoption of digital transmission formats allows 

30 media such as audio, video, and metadata content, to be associated, combined, and 

presented to provide viewers with a richer and more diverse media experience. Methods 
such as MPEG7 provide for relating content information, but do not provide a method by 
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which content of various foimats may be grouped, transmitted and displayed. Therefore a 
new method of organizing, tranamtting, and presenting media from multiple sources is; 
needed. 

Summary of the Invention 

The present invention overcomes the disadvantages and limitations ofthe prior 
art by providing a system and method that allows a transmission system to organize and 
transmit a related set of media and for a display platform to organize and render related 
media information in a manner that reflects the available media and the capabilities ofthe 
platform. A framework definition identifies a set of associated content (media) for a 
broadcast program. The present invention compares the format ofthe media with a 
transmission format and converts media of other formats to that of me transmission 
format. An oninimenu describes the content. Media content and the onmimenu are 
combined into a broadcast stream and transmitted. 

The present invention may Iherefore comprise a method for producing a broadcast 
stream that contains audio content, video content, and metadata content comprising: 
creating a framework defimtion that identifies the audio content, the video content and 
the metadata content associated with a broadcast and attributes thereof, comparing the 
audio format ofthe audio content with an audio transmission format and converting the 
audio content to the audio transmission format if the audio format and the audio 
transmission format differ, comparing the video format ofthe video content with a video 
transmission format and converting the video content to the video transmission format if 
the video format and the video transmission format-differ, comparing themetadata format 
ofthe metadata content with a metadata transmission format and converting the metadata 
content to the metadata transmission format if the metadata format and the metadata 
transmission format differ, creating a menu describing .the audio content, the video 
content, and the metadata content, combining the audio content, the video content, and 
the metadata content into a broadcast stream, transmitting the menu; and transmitting the 
broadcast stream. 
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A framework controller may utilize the framework definition to access media 
content to process and format the media content arid to control packaging and 
multiplexing of the content for broadcast 

The present invention may further comprise a system for combining multiple 
5 media and metadata streams having content into a framework for distribution of the 
. content to a viewer comprising: at J east one video source having an output, at least one 
audio source having an output, at least one metadata source having an output, a 
framework controller that receives the video source, audio source, and metadata source 
and produces an pmnimedia package integrating the outputs into a framework, a 
10 framework definition module that interfaces with the framework controller and defines 
all content to be used in the omnimedia package, a delivery module that receives the 
omnimedia package from the framework controller and transmits the omnimedia package 
to a receiver, and a receiver that receives and distributes the content of the omnimedia 
package to display devices and audio outputs, the receiver further coupled to at least one 
1 5 user input device that provides interactivity between the viewer and thereceiver. 

. ■ . . The present invention may utilize pre-loaded content that is transferred to a 
receiver prior to the broadcast of a media stream with which the preloaded content is 
associated. Pre-loaded content allows voluminous and complex content to be employed 
during a broadcast without requiring bandwidth to transfer the pre-loaded content at the 
20 time of broadcast and without latencies that may be incurred if the pre-loaded content 
were transferred at the time of broadcast 

The present invention may additionally comprise a method for rendering portions 
of a broadcast stream that contains audio content, video content, and metadata content. 
md a menu indicating the contents of the audio content, video content, and metadata 
25 content comprising: transferring preloaded metadata associated with the broadcast stream 
to a receiver prior to transmission of the broadcast stream, receiving the broadcast stream, 
displaying the menu wherein the menu includes an icon representing the preloaded 
metadata, receiving a user input, and rendering the preloaded metadata in response to the 
user input 

30 Advantageously, the present invention allows a viewer to select among a plurality 

of audio, video and metadata sources to obtain a television presentation tailored to the 
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viewer's preferences, offering increased viewing satisfaction. The present invention may 
be employed utilizing the capabilities of emerging digital transmission formats to provide 
an enhanced viewing experience and to increase audience size, allowing increased 
advertising revenue. 

Description of the Figures 

hi the figures, 

Figure 1 is a high-level block diagram of the present invention. 

Figure 2 depicts components of framework controller 108: 

Figure 3 depicts a receiver that may be employed with the present invention. 

Figure 4 is flow chart 400 of the operation of the metadata processor 214. 

Figure 5 is a flow chart 500 iUustrating the operation of omnimenu generator 224. 

Detailed Description of the Invention 

The present invention is directed to allowing the creation of larger, more robust 
productions with support for interactivity and multiple video, audio, and data streams. A 
framework definition provides organization of media for transmission, and for rendering 
of media by a display platform that may comprise a television, interactive television, set- 
top box, satellite receiver, personal computer or other equipment operable to receive data 
across a network or airwave and process data according to the method of the present 
invention. The framework definition allows a media stream or multiple media streams to 
be packaged together with other content into a single distinct program, hereinafter 
referred to as an omnimedia package. A content provider may employ the framework 
definition to specify and deliver a package of related content, encapsulating the 
information necessary to build, format, transmit and display the content. Content may 
comprise video, audio, and data information that may be streamed or cached. A wide 
range of information types and formats may be employed as illustrated by table 1 . The 
present invention is not limited to any specific types of information, formats, or 
relationships between information. 
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Table 1 



Media Tvpe 


Format 


CBR and VBR encoded video streams 


MPEG, PVB, etc 


Encoded Audio Streams 


MP2, MP3 . 


Packet-based encoded video/audio streams 


QuickTime, AVI, etc 


Program Synchronous Metadata 


ATVEF, Java, etc 


Scripted content 


HTML, Java, JavaScript, ECMAscript, VBScript, 
etc 


Interactive Graphical User Interface 


HTML, CSS 1, DOM 0 . 


Graphical Elements 


Logos, still-frame images, JPEG, GIF, etc 



The present invention maybe employed with broadcast systems that utilize 
terrestrial, cable, satellite, VDSL, or other transport methods- The present invention may 
also be employed with systems in which content is requested via a reverse path and with 
systems that support local or client side storage of pre-delivered content. Digital format 
transmission systems are well suited to the methodof ihe present invention* but analog 
systems with VBI data and Internet connection may also be employed. 

Figure 1 is a high-level block diagram of the present invention. A plurality of 
audio/visual arid metadata sources, which provide the audio, video, metadata and other 
data services, are combined to produce an omnimedia package. The term omnimedia 
describes the inclusive nature of the content being delivered by the present invention, as. 
contrasted with the usual audio/video packaging associated with television events. Thus, 
for a given event, the omnimedia package represents all of the synchronous and 
asynchronous components from video $ource(s)102, audio source(s) 104, and metadata 
sources 1 06, which may comprise and may be associated with a primary broadcast. Video 
source(s) 102 may comprise standard television broadcast channels (analog or digital 
modulation and encoding), any form of analog video (for example, Beta and VHS format 
tapes), any form.of stored digital video (for example, Laserdisc, DVD or any server- 
based digital video, such as may be used by video-on-demand systems) or may be any 
form of packet-based streaming media. This may include alternate camera sources, 
supporting video, alternate advertising video, localized or regionalized video. Audio 
source(s) 104 may comprise soundtracks that accompany one or more video source(s) 
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102, or may comprise an independent stream such as a radio signal or other audio only 
broadcast. Framework definition 1 10 defines the relationships between different content 
streams. For video source(s> 102 there may be zero, one or several associated audio 
source(s) 104. Further, a given audio source 104 may be assigned to zero, one ormore 
-video source(s) 102. There is no requirement that any given video source 102 have an 
accompanying audio source 104. Audio maybe encoded with a video source, such as 
may be practiced with packet-based streaming media. Metadata source(s) 106 may 
contain data packaged with respect to an event or associated with a general subject and 
may include executable code, scripts, graphics, logos, or general data; Metadata may be 
time synchronized, content related, or ancillary to an event 

Following are three examples illustrating the types and relationships between 
video source(s) 102, audio source(s) 104, and metadata source(s) 106, and how these 
sources may be organized and associated to provide a richer media presentation. In 
Example One, an omnimedia program is provided for the French Open, a tennis 
tournament featuring several simultaneous events. The primary broadcast may be from 
the main for center) court with accompanying broadcasts from other venues. 



Source ID 



102.LA 



104/1 .A 



104,1. B 



102J.B 



EXAMPLE ONE 



Source Type 



High bit rate video stream 
intended for full screen 
video. 



Audio Track 



Audio Track 



Low bit rate video stream 
intended for scaled video 
window. 



Source Description 



Video feed from center court, featuring players 

A and B: This is the Primary Video to be 
broadcast. This is the video that is broadcast 
to non-omnimedia enabled television systems. 
This is the English audio track for 102.1.A. 
This is the French audio track for 102. 1 .A. 



Camera focused on player A ? s coach and 
family members. 



102. LC 



102.1 L> 



102.1.E 



102. 1JF 



Low bit rate video stream 
intended for scaled video 
window. 



Camera focused on player B's coach and 
family members. 



Medium bit rate video 
■ stream. 



Medium bit rate video 
stream. 



Alternate camera angle 1 from behind player 

*~ a: : . 



Medium bit rate video 
stream. 



Alternate camera angle 2 from behind player 
B. 



Video feed from court 2, featuring players C 
• andD. 
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Source ID 


Source Type - 


Source Description . 


104.1.C 


Audio Track. 


" This is the English audio track fcir XCO- 1 V 


104. LD 


Audio Track 


This is the French audio track for 102,1 .F. 


. 10210 


Packet-based video stream 
with embedded audio 
track. 


Video arid audio feed from court 5; : featuring a 

• 111 1AVU Vl-V/ UL/Jw 1 H CLL\~>LX U \s t Vyv VI Jl IWU UvUllV. . „ 

. seeded pairs. . . \ „•' ".. 


106.1 A 


HTML Document 


Interactive page that shows statistics for match 
in progress, tightly synchronized to 102. 1 A 


106.1.B 


HTML Document 


Interactive page that shows summary of all 
102.1.A, 102.LB and 102.1 .G (video streams). 


106.1.C 

.-.i • " ' ' 


HTML Document - 


unci dvuvt?. pd.gt? uid.L snows resuiis. oi today .s 
matches and contains links to upcoming . 
matches Hink^ navip^tf* tr\ 1 Tfc sir* A *+' 

. 106.LE). .. 


106.1. D 


HTML Document 


Interactive page that provides detailed - 
commentary for an earlier match 


106.1.E 


HTML Document 


Interactive page that shows statistics and 
commentary for upcoming match 


106.1J? 


JPEGImage 


Logo image to be used by Interactive pages 
(106.1.A-106JJB);- 


Example Two shows the components of an ottnumedia package relating to the. 
fourth day of the Masters, a major golf tournament. 

EXAMPLE TWO 


Source ID 


Source Type 


Source Description 


102.2. A 


High bit rate video stream 
intended for full screen 
video. 


Video feed following, as appropriate and 
opportune, the various leaders, fhis is thV 
Primary Video to be broadcast This is tbe 

video that is broadcast to non-oiiMffiexlia 
enabled television systems/ : ' 


104.2.A 


Audio Track 


This is the audio track for 102.2 A- 


102.2.B 


High bit rate video stream 
- intended for full screen 
video. 


Video feed foUpwing Tiger Wood$. 


104.2.B 


Audio Track. 


This is the audio track for 102.2.B. 


106.2. A 


HTML Document 


Interactive-page showing Tiger Woods score 
card for the round in progress 


106.2.B 


HTML Document 


Interactive page showing leader board for 
tournament 


106.23c; 


HTML Document 


Interactive page showing past tournament 
winners 
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Source ID 



106.2.D 



Source Type 



Java Code 



Source Description 



Executable code that instructs consumer device 
■■ to show an animated view of the current hole 
in play on 102,2. A (loosely synchronized to 
102.2.A1 



Example Three shows the components of an bmramedia Da<-ka&e w Mn<, t«.« 
retransmission of the H. G. Wells "War of the Worlds^fbrJaS * * * 

EXAMPLE THREE 



Source ID 


Source Type 




104.3A 


Audio Track 


source Description 

Audio feed of the original broadcast. This is 
the Primary Audio to be broadcast. This is the 

video that is broadcast to nonH>mnimedia 
enabled audio systems such as an automobile's 
radio. 


104.3.B 


High Quality Audio Track. 


This is an audio track that has been redubbed 
and digitally remastered with added stereo 
sound effects. 


102.3.A 
104.3;C 


High bit rate video stream 
intended for full screen 
video. 
Audio Track- 


video, feed of an interview with H; G. Wells " 
about the making of and hysteria generated/by 
the original broadcast . " * - 


106.3A 


HTML Document 


This is the audio track for 102.3.A. 
Interactive page showing program 
synchronous factoids about me original 
broadcast of "War of the World". Tightly 
synchronized to 104.3%A, 


106.3.B 

L 


JPEG Image 


Program synchronous images showing photos, 
artwork and presi clippings related to 104,3a. 
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Framework definition 1 10 describes all content used in ari omnimedia package 
Framework controller 1 08 employs framework definition i 10 to create an onmimedi* I 



package. Framework defmition 110 defines the 



primary video and audio to be used.by ; 



non-onmimedia aware platforms, plus initial audio and video for ornnimedia package - 
stroams: Framework definition 1 1 0 is a Iso employed to generate a main-menu/omnimenu 
An ommmenu is aninteractive user interface that presents a menu of options available- 
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with an omnimedia package. Options may comprise different vide6 streams, different 
television layout, different audio sources, and other interactive data, streams. The 
definition of each stream may include a start/stop time to eiiable framework controller;" 
108 to switch streams during when creating the omnimedia package. An omniihenu may . 
5 be presented in a range of formats and may be customized by the system operator. For 
example, one embodiment of an omhimenii m ay scale current video into an upper right^ 
hand corner arid display a list of options to the left, of alternate video sources, alternate ■- 
audio sources, and links to other data. Another embodiment of an omiiiraemi may eipploy 
a pre-defined TV layout comprising a 3 A screen primary video format and several 
10 secondary thumbnail videos and data indicators across the bottom of the screen. 

Framework definition 110 may be created at a production facility or other site arid . 

may employ automated and manual methods of associating media. The framework; 

definition 1 10 may reflect limitations and constraints of source delivery systems, 

transmission systems, and display platform limitations. Such limitations may include 
15 * available bandwidth, number of channels tuned by a display platform, cache size, 

supported formats or other parameters. A framework definition rniay specify that images. 

must be in GIF format and that HTML code must support version 3.23, for example. For 

each event or presentation, a framework definition record may be produced that may . 

comprise information defining the type, size, nature, pricing, scheduling, and resource 
20 requirements for each media offered. Table 2 lists a number of components that may 

comprise a framework definition record. 

Table 2 

Framework Definition Record Components 
Version of the omnimedia package 

Oj/mimcau associated with the omnimedia package, may be created by framework controller. . 

Parameters used by framework controller 108 to allocate, control and prioritize transmission of stream elements. 

Flags/Gategory, languages, feature set; including mnfcipJe ratings available (e.g. PG thru R) 

Parameters employed to control delivery. (Tuning, URL, etc..) 

Price, if any: An associated cost for the omnimedia package 



9 



*>rr:in <wo o?4^qra? i > 



WO 02/43396 



O " ' ' ' 

PCT/US01/44510 



Time the bniDnneaia package expires * 

The number of framework definition stream records 

A plurality of stream records 



A stream record is provided for each media source (audio, video, or metadata) to be 
oifered in the bmnimenuL Each framework definition stream record may comprise: 
5 a unique ID; 

• a primary Y/N selection indicator, 

• the media Type (audio/video/data/other); 

• the bandwidth required for the media; 

• a start time/date time; 

• the date and time the omnimedia package expires; 

• a description having a paragraph or two describing the data stream (by 
way of example, an audio stream could include the person speaking/band, 
playing, a video stream could include the camera angle, or location, and a 
data stream could describe the interactive content and what it does. These 
are ns^d by the framework controller 108 when building the omnimenu; 

• a location (framework controller 108 would fill this out most likely) PID; . 

• flags category, languages, feature set, including multiple ratings available 
(e.g. PGthruR); • 

• price, including an associated cost for the stream as well as an additional 
fee beyond thfc cost of the program such that to see a certain rating version 
of a show or .to see video instead of just hearing audio may require a 
premium payment; 
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• . .; a unique Group ID within an Event that identifies a group of streams that 
V aire used by the framework controller 108 when building the omnimeriu 
. (note thiat only one stream from the group can be used at one time); and '." 

• any Dependent Streams, each having a unique stream ID which is required 
to access/use the stream. . hi use, a stream may require another stream. 
These , stream IDs may be used by the set top box to determine if the 
stream is available, i>e., an audio stream may be dependent on the video 
stream so that only devices that can view/use the video stream can appess, 
the audio stream. Also, the stream IDs can be used by head end equipment 
that is trying to optimize bandwidth and may want to separate different 
streams on different transponders for the same event 

An example of a framework definition is shown below: 

.» . 

' <X2UDL> 

. <Version>4</V ersion> 

<C^enu>Showme.html<^Menu> 

<Flags> 

<Language>English<yLanguage> 

<Rating>PG-l 3</Rating> 
</Flags> 

<Control> ???????? </Control> 
<Price>2,99</Price> 
<Expire>02/l 7/00 2 1 :00</Expire> 
<StreamRecords> 1 </S treamRecords> 
<SfreamRecordl> 

<3D>459812</ID> 

<Primary>True</Primaiy> 

<MediaType>Video</MediaType> 

<Bandwidth>256</Bandwidth> 

<Stait>02/15/00 22;00<?Start> 

<Expire>02/l7/00 21:00</Expire> 

<Descriptionx/Description> 

<Location>PID:200</Location> 

<Flags> 

<Language>English<yLanguage> 
<Rating>PG-l 3</Rating> 
</Flags> 

<Price>2.99</Price> 
<GroupID>002</GroupID> 
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<Dependan£>4598 1 0</Dependaht> 
<^StreamRecordl> . 

. </xsui> ... 

Employing framework definition 1 1 0, the framework controller 108 is operable to. 
format and organize media components into a stream or streams that may be delivered by 
the delivery system 1 12. Delivery system 1 12 may comprise a headend system and 
transmission apparatus such as employed by cable, satellite, terrestrial, and other 
broadcast systems. The organization and establishment of the stream(s) employs 
parameters provided in framework definition 110. The stream(s) is (are> delivered to 
receiver 1 14 that processes the stream(s) and provides output to display device(s) 116, 
audio output(s) 118, and may send and receive signals to/from user input devices) 120. 

Figure 2 depicts componen ts of framework controller 108. Framework controller 
108 includes framework control logic 216 that is operable to retrieve and interpret 
framework definition 110, and employ the parameters thereof to control operation 
preprocessors 210, 212, and 214, plus packagers 218, 220, and 222 to format and 
encapsulate information (video, audio and metadata) for multiplexer 226. Video 
preprocessor 210 and audio preprocessor 212 access media streams or stored data as 
specified by framework definition 1 10 and perform processing to prepare the media for 
the associated packager. Such processing may include rate adaptation, re-encoding, 
transcoding, format conversions, re-sampling, frame decimation, or other techniques and 
methods to generate a format and data rate suitable for packagers 218, 220, and 222 as 
may be specified by framework definition 110. Processing may include MPEG encoding 
of analog video as may be supported by encoder equipment from Divicom Inc. of 
Milpitas, CA., which is a wholly owned subsidiary of C-Cube Microsystems, Inc. When 
such processing is specified, framework controller 108 may provide a sequence of 
instructions to the encoder, selecting channels and controlling encoding. 

Metadata preprocessor 214 accesses metadata elements, specified by framework . 
definition 1 10, and performs processing to prepare these the metadata packager 222v ' 
Such processing may include transcoding^ format conversions, script conversions, script 
generation, and image format conversions, for example. In operation, graphical metadata 
maybe sent to metadata preprocessor 214 in a computer graphics format (Photo Shop, for 
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example) that then may be converted to a format that the display platform recognizes (gif, 
for example). 

Figure 4 is flow chart 400 of the operation of the metadata processor.214. At step 

: 402 an image file is accessed by metadata preprocessor 214. The metadata preprocessor . 
5 accesses metadata and places it in a predetermined format as may be specified by 
framework definition 110. Metadata may comprise graphics data, sound data, HTML 
data, video d&ta; or any other type of data. Metadata preprocessor 214 processes the data * 
arid outputs the processed data in a transmission/multiplexer format. As shown in figure 
4, the flow diagram allows the image file to he converted and output by the metadata * 

10 processor 2 1 4 in real time. At step 404' the graphic file conversion defini tions are loaded ; 
into the metadata preprocessor 21 4 to perform the conversion of the image file. At step 
406 the image file 402 is converted into the graphic file in accordance with the 
definitions. At step 408 the metadata preprocessor 214 outputs the converted image file 
to the metadata packager 220. A high-speed processor may perform these functions in 

15 real time using definitions that can be quickly downloaded from a high-speed storage 
device. Custpm designed state machines may also be employed for format conversion.; 
Metadata preprocessor 214 may also be employed to convert the format of HTML 
metadata. HTML metadata maybe sent to metadata processor 214 in HTML 4.0 format 
that then may be converted to HTML 3.2 format such as may be required by a display 

20 platform. The framework controller generates commands that are sent .to metadata * . . 
processor 2 1 4 that identify the metadata and specify the output format of the metadata. 

Referring again to figure 2, video packager 218 and audio packager 220 are 
subsystems that package video and audio assets into a format that is compatible with the 
. delivery system 1 12, including for example, packetization, serialization* rate control, 

25 null -packet insertion, and other functions required to prepare the video for compliant 

transport via MPEG, DVB-C/S/T, PacketVideo, or other transport fdnnats. For example, 
General Instruments (now owned by .Motorola) produces a QAM modulator that . 
modulates MPEG video that is encoded from video preprocessor 210 to digital video 
tooadcast (DVB) format; The frameworic controller generates commands that are sent to 

30 the QAM modulator specifying the frequency (channel) and PID (packet identifier) for 
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the video based upon the framework definition that was provided for the omnimedia 



As also shown in figure 2, metadata packager 222is a subsystem that performs 
• packaging on all metadata elementsthat are to be included in the omnimedia package. 
Metadata packaging may comprise rate control, packetization, serialization, and 
synchronization to video and/or audio streams. The metadata is also prepared for • * 
transport across any compliant transport mechanism (MPEG, DVB.-C/S/T, PacketCable, - 
DVB-MHP, etc.)- A commercially available product for performing these functions is 
the TES3 that is provided by Norpak Corp., Kanata, Ontario. The TES3 encoder encodes 
metadata into a NTSC signal with NABTS encoding. NABTS is the protocol that allows 
metadata to be sent in me VBIfvertic^ The 
framework controller commands the TES3 encoder as to what lines of the VBI are, . 
employed for transmitting metadata. 

Referring again to figure 2, omnimenu generator 224 may be implemented as a 
rules-based subsystem employing framework definition 208 to generate a user-interface 
that presents. program options and allows a viewer to select from these options. Rules 
maybe employed to generate an omnimenu template in HTML page format. The HTML 
page may comprise a full screen image containing buttons (active icons) that may be 
selected to activate a particular media stream. Omnimenu generator 224 employs . , 
framework definition 208 to identify available streams and associates each stream with a 
button. 

•.. ; . Figure 5 is a flow chart 500 illustrating the operation of omnimenu generator 224. 
At step 502, omnimenu generator 224 accesses the framework definition 1 10. At step 
504, PIDS (program IDs) are extracted from the framework definition. The omnimenu 
template, described above, is loaded into the omnimenu generator 224 at. step 506 of 
figure 5. At step 508, each of the PIDS that have been extracted from the framework 
definition are assigned to a button such that each of the buttons is labeled with the PID 
name. At .step 510, selection functions are assigned to each of the buttons in accordance 
with the labels that have been assigned to those buttons. In this manner, the video can be 
changed in accordance with the labeled functions of each of the buttons of the template. 
At step 512 the template maybe assigned a company logo. In this manner, the content 
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may be properly braMed to correspond to the source of that content At step 514 the. 
omnimenu generator 224 saves or exports the new omnimenu. The omnimenu may be 
exported to the package multiplexer 226. 

... As shown in figure 2> package multiplexer 226 combines the package elements 
5 into a stream or set of streams, in preparation for transmission. The streams are coupled 
through delivery system interface(s) 228, to delivery system 1 12, and the physical and 
logical interconnects to the transmission system. A Divicom, Inc., (Milpitas, CA) MUX 
is an example of a package multiplexer, having the different frequency and PIDs. 

As indicated in figure 1 > delivery system 1 12 then transports the streams to the 

10 receiver 114. Delivery system 112 may be analog or digital. In general, system delivery 
of content is as open as needed for any particular package and topology. A simple movie 
with no added content may be delivered on a single broadcast multiplex, while a package 
for the Super Bowl would contain many different delivery mechanisms. Some may be 
available on a particular receiver and not on others. * 

15 The omnimenu may provide a directory °f available media for a broadcast event. 

The omnimenu is transmitted to a plurality of receivers 1 14. Receivers 1 14 may vary in 
capability and may include upstream (reverse path) communication from the receiver to 
the headend system, or may use other return systems such, as an Internet connection^ for 
example. Receivers 114 that do not include upstream communications may employ the 

20 omnimenu to select audio, video and metadata information contained in a broadcast 
stream. The bandwidth of an analog NTSC channel may be employed to carry several 
digital video streams, audio streams, and metadata. The omnimenu includes tuning or 
packet information to identify streams that may be accessed by the receiver. The receiver 
; includes a software program operable to display the omnimenu and operable to tune and 

25 render selected streams. 

In another embodiment of the present invention, receiver 1 14 supports upstream 
communications. The headend system, in response to upstream communications, may 
provide on-demand programming and data delivery. The omnimenu initiates upstream 
communication in response to user selection of a displayed media button (icon). The . 

30 headend system may supply the requested stream in a broadcast channel, or if already 
broadcast, may provide tuning or packer decode information to the receiver. The 

i. . 
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. requested stream may be employed to v^date&eJtoetvoTfc definition 110. Framework 
controller: 108 uses the framework definition 1 10 to allocate bandwidth and PID ■ 
information. The framework controller provides the frequency, PID and bandwidth" - 
information from the framework definition and uses it to send/control video and audio 
packager 218/220: Video packager 218 and audio packager 220 (figured) then allocate 
bandwidth for each of the respective video streams. . 

As also shown in figure 1, information from the framework controller 108 is . 
. encoded into the omnimenu so that the receiver will be able to tune to and decode the 
streanis. The framewoikcontroller 108 may also include URL's for demand data or. : 
streaming media. Locations based on alternate tuner systems may also be included. For 
example, a radio station frequency having local commentary may be simulcast with the 
video. Depending of receiver 1 14 capability, some data and/or programming may be 
loaded in advance into storage built into the receiver such that the content is available 
locally for viewing during the airing of the primary content package. For example, all 
NFL player statistics may be preloaded over a trickle feed before the Super BowL In this 
manner, an interactive fantasy football application may retrieve all needed statistics 
during the game in order to let the viewer play a fantasy game during the airing of the 
primary program. 

Figure 3 depicts a receiver that may be employed in accordance with the present 
invention. Receiver 302 is used by the system end-user or viewer for use in viewing and 
controlling the oinnimedia package. The receiver 302 may comprise a decoder 304, 
parser 306, media controller 308, receiver cache 310 and receiver controller 312. Decoder 
304 may extract framework information from the delivery stream omnimenu. Infonnation 
may be. encoded in the vertical blanking interval. Decoder 304 may comprise a NABTS 
VBI decoder from ATI Technologies, Inc., of Thornton, Ontario. The VBI decoder may 
extract data from the VBI and present it to framework parser 306. The data may comprise 
an XML format file of the oniriimenu. Parser 306 extracts elements mat comprise the 
framework. Framework parser 306 receives framework data from decoder 304 and- 
prepares the data for use by me receiver controller 312. The receiver controller 312 may 
comprise a data interpretation module Media controller 308 selects the media streams 
(audio, video and/or data) that are. described by the fiamework definition. The media 
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controller comprises 308 a tuner and video/audio decoder. The media controller 308 . 
receives stream data from decoder 304 and control signals from receiver controller 312, 
Media controller 308 selects the proper media PIDS as needed and feeds them to display ~ 
device(s) 1 1 6, audio device(s) 1 1 8, or user input device(s) 120. Receiver controller 312 . 
5 serves as the central processor for the other elements of receiver 302; The receiver 

controller 312 may compare the capabilities of receiver 302 with the stream type to; - \ 
detemiipe which streams may be received and used by receiver 302: Receiver controller 
3 12 sends control signals to the other units in the system; The functions of recei ver ~ : 
controller 3 12 may be performed by software resident in the control CPU of the receive* 
10 302. Functions include receiving data from framework parser 306 and signaling media - f 
controller 308, For example, selection of omnimenu items may alter the framework such I 
that the receiver controller 312 receives infonnation from framework parser 306 and; . f 
signals media controller 308 to render the selected media stream. Receiver cache 310 

may be employed to store the framework definition and any parameters, code objects, [ 
15 data items or other software to control omnimenu display and response to, user input. .. 

r Receiver 302 receives data associated with a content package and employs the - 
data to access the package contents. Advanced receivers may check system capabilities to 
determine which pieces of content may be rendered. For example, a digital set top box 

may be able to decode MPEG video and audio. A digital set top box may also be able to V 
20 decode MP3 .audio, but not be able to decode HDTV signals. A radio type receiver-may 
only be? able to decode, audio formats and, possibly, only formats not related to a video 

signal. The receiver may also be able to decode various data types from data services, : . § 

These rnay take the fonn of application code that would be executed on the receiver if I 

compatible. The receiver of the present invention has the internal compatibility to be . f 

15 able to receive the omnimedia packaged signal and decode the parts that are relevant to * 

its capabilities to give the fullest experience it can, regardless of the format i 

... Referring again to Fig. 1 , display device(s) 116 provide(s) the visual presentation [ 
*■*•*•- - t; 

pf .the framework to the end-user/viewer, including video, streaming media, a graphical | 

user interface, static graphics, animated graphics, text, logos and other displayable assets.- 
10 Similarly, audio outpyt(s) 1 1 8 presents) the audio portions of the framework to the end- 
user, including primary audio, possible secondary audio tracks, streaming media, audio 
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feedback from the receiver 1 1 4, and other audio assets. User input device(s) 120*llow(s) 
the user to control the receiver and interact with the framework, providing the ability to 
choose components of the framework, select links within the framework, navigate the 
graphical user interface of the receiver, or perform other interactions with the receiver. 

The foregoing description of the invention has been presented for purposed of 
illustration and description. It is not intended to be exhaustive or to limit the invention to 
th e precise foraCdisclosed, and other modifications and variations may be possible in V 
light in the above teachings. The embodiment was chosen and described in order to; best 
explain the principles of the invention and its practical application to thereby enable: ' : 
others skilled in the art to best utilize the invention in various embodiments and various 
modifications as are suited to the particular use contemplated. It is intended that the 
appended claims be construed to include other alternative embodiments of the invention 
except insofar as limited by the prior art. 
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Claims 

What is claimed is: ' ... " " 

h - A method for producing a broadcast stream that contains audio content, video : . - 
contmt, arid metadata contem comprising: 

creating a framework definition that identifies said audio content, said video 
content and said metadata content associated with" a broadcast and attributes thereof . 

comparing the audio format of said audio content with an audio tr^ fc > 

format and converting said audio content to said audio transmission format if said . % / 
V. audio format and said audio transmission format differ; 

• % . comparing the video format of said video content with a video transmission 

. format and converting said video content to said video transmission format if saidy 
video format and said video transmission format differ; .; . - 

' comparing the metadata format of said metadata content with a metadata / 
■ " : . transmission format and converting said metadata content to said metadata 
' transmission format if said metadata format and said metadata transmission format 
<tiffer;.and 

^^bining said audio content, said video content, and said metadata content 
. - . into a broadcast stream, 

2. The method of claim I wherein said framework definition further comprises: ... 

a fiameworic definition record for each element of said audio content, said 
video content, and said metadata content r , 

3. . The method of claim 1 wherein said menu further comprises: 

•v \/\±p an icon for each element oif said audio content, said video content, and' . 
said metadata, content. 
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4. - The niethod of claim 1 wherein said metadata content comprises an image file. 

5. The method of claim 5 wherein said converting -said metadata content further 
comprises: \ 

. - loading said image file; . 
loading a . file c^hVj&rsion definition; : 
converting said file using said conversion definition , ah 
oatputting ;a converted image file. 

S- The method of any one of the preceding claims further comprising the steps of: 

. creating a menu describing said audio content, said video content, and said 
metadata content; 

: transmitting said menu: and 

* . * " i *.."**. " 

. .transmitting said broadcast stream. 

7. A method for rendering portions of a broadcast stream that contains audio content, 
video content, and metadata content and a menu indicating the contents of said audio, 
contend video content, and metadata content comprising: 

transferring preloaded metadata associated with said broadcast stream to a 

* receiver prior to transmission of said broadcast stream; 

receiving said broadcast stream; 

displaying said menu wherein said menu includes an icon representing 
said preloaded metadata; 
receiving a user input; and 

rendering said preloaded metadata in response to said user input. 



v. 
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8. A system for combining multiple media and metadata streams having content into 
a framework for distribution of the content to a viewer, comprising: 

a framework controller that receives said video.source, axjdio source, and 
* * metadata source and produces an omnimedia package integrating said outputs into a 
framework; and 

a framework definition module that interfaces with said framework . 
* controller and defines all content to be used in the omnimedia package. 

9. The system of claim 8 further comprising: 

at least one video source having an output; 
at least one audio source having an output; 

at least one metadata source having an output* . . 

10. The system of claim 8 or 9 further comprising: 

2C delivery module that receives said omnimedia package from said 
fraftiew;ork controller and transmits said omnimedia package to a receiver; and 

a receiver that receives and distributes the Content of said omnimedia 
- package to display devices and audio outputs, said receiver further coupled to at least 
one user input device for providing interactivity between said viewer and th^ receiver. 
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